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[57] ABSTRACT 
A number of intelligent bus interface units (100) are 
provided in a matrix of orthogonal lines interconnecting 
processor modules (110) and memory control unit 



(MCU) modules (112). The matrix is composed of pro- 
cessor buses (105) and corresponding control lines; and 
memory buses (107) with corresponding control lines 
(108). At the intersection of these lines is a bus interface 
unit node (100). The bus interface units function to pass 
memory requests from a processor module to a memory 
module attached to an MCU node and to pass any data 
associated with the requests. The memory bus is a pack- 
et-oriented bus. Accesses are handled by means of a 
series of messages transmitted by message generator 
(417) in accordance with a specific control protocol. 
Packets comprising one or more bus transmission slots 
are issued sequentially and contiguously. Each slot in a 
packet includes an opcode, address, data, control, and 
parity-check bits. Write-request packets and read- 
request packets are issued to the memory-control unit 
The memory-control unit responds with reply packets, 
A message controller (416), bus monitor (413), and 
pipeline and reply monitor (414), run the memory bus in 
a three-level pipeline mode. There may be three out- 
standing requests in the bus pipeline. Any further re- 
quests must wait for a reply message to free-up a slot in 
the pipeline before proceeding. Request messages in- 
crease the length of the pipeline and reply messages 
decrease the length of a pipeline. A control message, 
called a blurb, does not aflect the pipeline length and 
can be issued when the pipeline is not full The different 
messages are distinguished by three control signals 
(405) that parallel the data portion of the bus. The mes- 
sage generator (417) and interface logic (404) drive 
these control lines to indicate the message type, the start 
and end of the message, and possible error conditions. 
The pipeline and reply monitor (414) and the message 
controller (416) cooperate to insert a reply to a particu- 
lar request in the pipeline position corresponding to the 
particular request that invoked the reply. 

8 Claims, 3 Drawing Figures 
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DETAILED DESCRIPTION 

FIG. 1 is a diagram of a data processing system em- 
ploying an interconnect mechanism in which the pres- 
ent invention is embodied. The interconnect mechanism 5 
provides data processors with connection and access 
capability to a main memory. The interconnect mecha- 
nism is more fully described in the above-referenced 
U.S. Pat. No. 4,438,494, "Method and Apparatus of 
Fault-Handling in a Multiprocessing System " by David 10 
Budde et al> granted on Mar. 20, 1984 and assigned to 
Intel Corporation. 

There are three module types in the central system: 
GDP, IP (with an associated attached processor, AP), 
and Memory. All GDP modules are identical; thus, any 15 
GDP module can act as a backup resource for any other 
GDP. Each IP module provides an interface between 
the central system and one I/O subsystem. All IP mod- 
ules have identical interfaces into the central system, 
but potentially different I/O subsystem interfaces. To 20 
act as a backup resource, the IP module must interface 
to the same AP. Memory modules are only attached to 
two memory buses, the primary and backup buses. 
Backup memory modules must be attached to the same 
two memory buses. 23 

The interconnect mechanism is comprised of two 
Very Large-Scale Integrated-circuit (VLSI) chips. One 
chip is a bus interface unit (BIU-100), and the other chip 
is a memory control unit (MCU-102). The interconnect 

mechanism is a system of orthogonal lines arranged in a 30 basic communication of requests and data between the 



single 256K byte to 4M byte dynamic memory array. 
As access requests are passed along the memory bus, the 
MCU selects and responds to those requests directed to 
addresses within the address space of its array. 

Since GDP processors make variable-length accesses 
on any byte boundary, single access will sometimes 
span two memory buses. A time-ordered memory bus 
arbitration scheme described subsequently guarantees 
access indivisibility. 

Each MCU in the system logically attaches to one of 
the memory buses (M ACD) and controls a single four- 
byte- wide array of memory. 

The memory storage interface generated by the 
MCU drives a memory .module (112) which is com- 
prised of a single 40-bit array (includes 7 ECC bits and 
a spare) of memory (256K bytes using 64K RAM parts). 
The memory interface contains the following signals: 
SLAD bus (519): 20-bit multiplexed address and data 
(latches are provided next to the array to latch and 
multiplex the address for multiplexing from/to the 
20-bit bus); 
Control lines (521): 

RAS, CASIN (dynamic RAM timing); 
WE: write Enable; 

DE1N: Address Mux (selects row or column address 

in an external address multiplexer) 
The bus interface unit/memory bus is a packet-ori- 
ented bus. A packet is an identifying "envelope" or 
header plus message contents (data) which provides 



matrix. This matrix is composed of processor buses 
(ACD) and corresponding control lines, and memory 
buses (MACD) and corresponding control lines. At the 
intersection of these lines there is a bus interface unit 
(for example. 100). The bus interface unit responds to 33 
access requests from a processor (HO) received over the 
processor bus, and switches them onto the appropriate 
memory bus for distribution to the correct segment of 
the address space in a memory module (112). Similarly, 
reply messages from a memory module are routed onto 40 packet bus: 



MCU and the bus interface units. The packet bus con- 
tains twenty-one signal wires which are shown in FIG. 
1: 

MACD (15:0)= 16 bits of multiplexed data, address 

and control 
CTL (2:0)= 3 bits of packet control 
CHK (1:0)=2 bits of parity 

Five arbitration signal wires are provided to arbitrate 
among bus interface units contending for access to the 



the memory bus by the Memory Control Unit (MCU- 
102). In response to a reply message, the appropriate bus 
interface unit selects and switches the data onto the 
processor bus in correct sequence. 

Each processor bus supports one Generalized Data 45 
Processor (GDP-110) or one I/O Processor/Attached 
Processor (IP/AP) pair (111). The processor bus is a 
synchronous, multiplexed, bidirectional, 16-bit bus. 
This bus makes connection to the interconnect system 



on the processor side of the bus interface unit (100). The 50 al U.S. Pat. No. 4,438,494. 



NREQOUT 
RQOUT 
RQ# 
CONT# 
NREQ# 

Appropriate response to hardware error conditions is 
based upon a "confinement area" concept which parti- 
tions the interconnect system of FIG. 1 into a number of 
areas, shown in detail in the above- referenced Budde, et 



processor bus is described in detail in the above- 
referenced U.S. Pat. No. 4,315,308 "Microprocessor 
Interface Control Apparatus" of Daniel K. Jackson, 
filed Dec. 21, 1978 and assigned to Intel Corporation. 
The GDP (110) is described in detail in the above- 
referenced U.S. Pat. No. 4,325,120 "Data Processing 
System" of Stephen Colley et al, filed on Dec. 21. 1978 
and assigned to Intel Corporation. The IP (111) is de- 
scribed in detail in the above-referenced U.S. Pat. No. 



The signal lines that make up the interfaces between 
the various components comprising the interconnect 
mechanism are briefly described in the paragraphs that 
follow. 

55 The following signal lines allow communication be- 
tween one attached processor and bus interface units in 
a column of the switching matrix. A bus interface unit 
reads initialization and configuration information on 
these lines at cold start time (1NIT# assertion). A bus 



4,407,016 "Microprocessor Providing an Interface be- 60 interface unit may signal reception of an interprocessor 

tween a Peripheral Subsystem and an Object-Oriented communication (IPC) sent to the processor in accor- 

Data Processor," of John A. Bayliss et al, Hied Feb. 18, dance with the teachings of the above- referenced U.S. 

1981 and assigned to Intel Corporation. Pat. No. 4,315,308. Electrically, there are four signal 

Each memory bus supports one or more memory types employed: input only, output only, bidirectional, 

modules (112). Each memory module is connected to 65 and wire-or. The suffix indicates that the line is 

the memory bus through the Memory Control Unit effected when asserted low. The wire -or signals use pin 

(MCU, 102). The MCU works together with the bus pairs: a line driver and an observer. This way, a compo- 

interface unit in routing accesses. Each MCU controls a nent may assert one of these signals and all others may 
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observe. The line driver (output) drives an open -collec- 
tor Schottky TTL inverter, which acts on (pulls down) 
the signal line (an input). The convention used is to 
name the signal line (and input pin) by the function of 
the line. (An external pullup is required on these lines 5 
and they are asserted low.) The line driver (output pin) 
has 'OUT appended to the basic name. An example of 
this is the RQ#-RQOUT pair. 
ACD Bus [15:0]-(bidirectional) 
The Address/Control/Data (ACD) lines are the 1° 
basic information path between the bus interface 
units in a column and the processor in the same 
column. Typical operation involves time-multi- 
plexing data on these pins in the following order: 
Control information for bus transactions. 
Address information for the data operation re- 
quested, 

Data requested by the preceding address and con- 
trol. 

The interpretation of data presented on ACD 15:01 is 
controlled by ISA and I SB, described below. Also, 
certain configuration information is accepted from 
the ACD 15:01 pins, with assertion of INIT#. 

ISA(input) 

Initiation of a transaction by a processor is signaled 
using ISA as a data latch enable. Also, cancellation 
of a processor's request may be signaled on ISA, by 
stretching its assertion an addtional cycle. 
ISB(input) 

To allow cooperation between bus interface unifs on 
a processor bus, to control data latching times of 
the attached processor, and to inform a bus inter- 
face unit about the condition of another's transac- 



ts 



20 



25 



50 



As a new time-ordering service period is recognized, 
an arbiter machine in a bus interface unit will assert 
request (RQOUT) if it has a request pending for 
participation in arbitration in that time-ordering 
cycle. All the RQOUT lines from bus interface 
units in a row (parallel to the MACD bus) are 
or-tied to the RQ# line (an input described below). 
Any one RQOUT asserted will pulldown RQ#. 
NREQ#-(input, low asserted) 
NREQOUT lines from bus interface units in a row 
drive open-collector inverters, which act on the 
signal line NREQ# and cause it to be pulled down 
when any one or more NREQOUTs are asserted. 
New time-ordering cycles are recognized by bus 
interface units when NREQ# is asserted. 
CONT# finput, asserted low) 
RQOUT lines from bus interface units in a row drive 
externa) circuits, which act on the signal line 
CONT# and cause it to be pulled down when any 
two or more RQOUTs are asserted by the arbiter 
machines within the bus interface units. CONT# 
indicates that there is contention for the bus. 
RQ# (output, asserted low) 

RQ# is an input from an external line driven by all 
the or-tied RQOUT lines from the modules (bus 
interface units) attached to the MACD bus. Any 
one RQOUT asserted will pull down RQ#. As a 
new time-ordering service period is recognized, a 
arbiter machine in a bus interface unit will assert 
request (RQOUT) if it has a request pending for 
participation in arbitration in that time-ordering 
cycle. 

The following signal, lines allow a bus interface unit 
to communicate with memory units and other bus inter- 



tion(s), the ISB pin is provided to sample the wire- 3J f acc „, a row on its attached memory bus. Data, 



data check, and bus and buffer control lines are pro- 
vided. A bus interface unit reads some initialization and 
configuration information on these lines at cold start 
time (INIT# assertion). 
40 MACD [15:0] (bidirectional, tristate outputs) 

The primary data path between the bus interface unit 
and attached memory modules is the MACD bus. 
Typical operation involves time-multiplexing data 
on these lines in the following order: 
Control information for bus transactions, 
Address information for the operation requested. 
Data requested by the preceding address and con- 
trol. 

Three signal lines, CTL [2:0], described below, pro- 



or state of all of the bus interface unit's ISBOUTs. 
ISB assertion changes its meaning as an ACD 
transaction proceeds: IPC, data stretch, data good, 
bus error, as more fully described in the above- 
referenced Jackson patent application. 

ISBOUT-<output) 
Response to processor requests is signaled using IS- 
BOUT as a data latch enable and an error indicator 
(following the transaction). Also, when no transac- 
tion is underway, the bus interface unit may signal, 43 
on ISBOUT, reception of an Interprocessor Com- 
munication message (IPC) to the attached proces- 
sor. The ISBOUT lines from bus interface units in 
a column (parallel to the ACD bus) drive open-col- 
lector STTL inverters, which act on the ISB signal 50 vide timing and validity information for data presented 
line, on the MACD bus. Stretch cycles are not supported on 

The following signal lines allow a bus interface unit the MACD bus by the bus interface unit; requests and 

to share access to the memory bus with other bus inter- acknowledgments are transmitted as full, contiguous 

face units in a row. These signal lines allow the handling blocks. Certain configuration information is accepted 

of a new processor request, gaining memory bus access, 55 from this bus with assertion of !N1T#. 

managing time-ordered arbitration, and cooperation CTL [2:0] (bidirectional, tristate outputs) 

with other bus interface units. 

NREQOUT(outpuU asserted high) 
Upon recognition of an access request from a proces- 
sor requiring use of the memory bus, a bus interface 60 
unit signals existence of a new time-ordering cycle 
to other bus interface units by asserting NREQ- 
OUT. NREQOUT lines from bus interface units in 
a row (parallel to the MACD bus) drive open-col- 
lector inverters, which act on the signal line 65 
NREQ# (below) to cause It to be pulled down 
when any one or more NREQOUTs are asserted. 

RQOUT(output, asserted high) 



. These signals denote the start and end of the three 
types of memory bus transactions: requests (op- 
coded), replies (nonopcoded, implicitly addressed 
reply), and blurbs (opcoded). Any memory bus 
component drives these signals with data it asserts 
onto the memory bus. The encoded meanings of 
the signals are: 
111. Idle 

001 Request: normal 
000 last cycle warning 
011 Reply: normal 
010 last cycle warning 



4,480,307 

7 .8 

110 first cycle of write ack. nondata reply (last WE# (Output) 

cycle signif.) Write strobe used for write cycles and correcting 

101 Blurbs: normal (not presently used) scrub cycles. 

100 last cycle warning DEIN (output) 

The following signal lines allow communication be- 5 External latch data enable (Master mode) 
tween one attached memory control unit (MCU) and CASIN (input) 

the bus interface units in a row of the switching matrix. CAS strobe FRC input (checker mode) 
The signal lines described below provide system infor- INTERFACE UNIT NODES 

mation to the MCU, defining its behavior in a data 

processing system: 10 The primary function of the bus interface unit (100, 

CLKA, CLKB (inputs) 102, etc.) of FIG. 1 is to pass memory requests from a 

CLKA and CLKB provide basic timing references processor, (110 or 111) to an attached memory module 
for the B1U and MCU. CLKB lags CLKA by (112) and to return to the processor any requested data, 
(nominally) \ cycle (90 degrees). Arbitration re- Referring now to FIG. 2, the bus interface unit com- 
lated signals are driven with CLKA falling, and 15 ponent (100) is composed of a number of state machines, 
sampled with CLKA falling. MACD [15:0] is the functions of which will now be described briefly, 
driven and sampled with CLKB rising ($ cycle The ACD bus interface logic (402) receives and de- 
timing). Buffer directional control for the memory codes requests from the processor. A transaction start is 
bus is altered with CLKA rising. signaled by means of the ISA line from the processor as 

INIT# (input, asserted low) 20 more fully described in the above-identified Jackson 

System components respond to INIT# by resetting patent application, 
to an initial state, then waiting for requests by ex- The switching logic (406) includes means for buffer- 
ternal components. ing data transmitted between processor and memory. 

The following arbitration/handshake signal tines As data comes in for a write request, it is deposited here 
allow an MCU to share access to the memory bus with 25 waiting for memory bus access and issue. As data comes 
other memory bus components: in from a read acknowledgment, it is also deposited 

RQ# (input, asserted low) . here, waiting for return to the processor. 

RQ# is asserted when there are one or more The time-ordered queue (408) separates requests into 
RQOUTs asserted. time-order slots and assures that all requests in a particu- 

CONT# (input, asserted low) 30 lar time-order slot are serviced before any requests in 

CONT# is asserted when there are two or more the next time-order slot The time-ordered queue also 
RQOUTs asserted. This indicates that there is con- generates the internal line ARBREQ to the arbiter ma- 
tention for the bus. chine (410) to submit a received request to arbitration. 

The following memory bus signal lines allow an The arbiter machine (410) controls the energization 
MCU to communicate with bus interface unit nodes on 35 of NRQOUT which causes other bus interface units to 
its attached memory bus. An MCU reads some initial- be informed of a new processor request on the NREQ# 
ization and configuration information on these lines at signal. Assertion of this signal causes ail bus interface 
cold start time (INIT# assertion): units to queue up a new time-ordering cycle. As re- 

MACD (15:01 (bidirectional, tristate outputs) quests are satisfied the bus interface units will dequeue 

See description above. 40 cycles. The arbiter machine (410) In each bus interface 

CTL [2;0] (bidirectional, tristate outputs) unit performs this protocol, allowing each processor 

See description above. equal weight in obtaining memory. The bus access right 

BUSSEL (Output) is signaled by the grant line to the grant queue (412). 

This signal line allows an individual MCU chip to be Controlled access to the MACD bus and the issuing 
enabled onto one of two buses. This may be used at 45 of request and reply messages is performed by the pipe 
initialization time to help configure the memory monitor (414), the message controller (416), and the 
modules on the bus interface unit buses. The selec- MACD bus interface logic (404). 
tion of one or the other buses is done by writing Recognizing requests, a bus interface unit will inform 
into the State Register in the MCU's local address other bus interface units of a new processor request on 
spacc . 50 the NREQ# signal line by asserting NREQOUT. As- 

The following array interface signal lines provide the sertion of NREQ0 causes all bus interface units to 
control of the array or memory module side of the - queue up a new time ordering cycle. Satisfying the 
MCU: requests of each time-ordering cycle, all bus interface 

SLAD (19:0) (bidirectional, tristate outputs) units will dequeue cycles. Time-ordering circuitry in 

This is the main bus for transferring information to or 55 each bus interface unit performs this protocol, allowing 
from the array. During a memory access the each processor equal weight in obtaining memory. 
SLAD bus is used for the following types of data Initialization and Configuration: At system reset time, 
transfers. the bus interface unit must be supplied with enough 

Memory Address information to define its functionality in the system 

Read Data 60 environment. During assertion of INIT# (cold reset 

Write Data (in the case of a write access or a cor- time), initialization and configuration information is 
reeling scrub) brought on chip. 

RAS (Output) Thc following processor specific information is pro- 

Determines when an array access will start and when vided with INIT#: 
it will end. It also is the Row Address Strobe ad- 65 Assertion (cold reset) over the processor bus: 
vanced so that the actual RAS# to the array can be . Processor bus number (module ID along processor 
clocked ofT of an external flip-flop to produce well axis): 6 bits 

controlled timing. Processor type: 3 bits 
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To establish addressing coordinates for individual bus request (processor ."StaKTta 
interface units in the buT interface unit interconnection accesses include the write data and increase bus 

scheme, each processor bus is assigned a unique number pipe depth). , „„„ . 

S SS (INIT#) time. The number* cannot be reply (any response to a request; repl.es decrease the 

dress for the bus interface unit component. The other sage, ana nows ior wc siuuic, 

tion. enl messages are differentiated by three control signals, 

MEMORY CONTROL UNIT (MCU) cTL [2:0], that parallel the data portion of the bus. The 

n.f„™„ tn FIG 3 the memory-control unit (102) 20 message source (bus interface unit or MCU) drives the 
shownrilO "wul now Sm. The memory control lines to indicate the message type, the start and 

faceportlon and a storag, • "rface port.cn. sepa- All «en ^of he bus. ^ & ^gc 0 n the 

is provided to control the msemon of a reply message «BWM» ^ ^ addresses ^ multiplexed . 
^sZjeTJ tterfacc include a request FIFO 35 ^ busing ^f =e ^^tcS 

drive an SLAD bus interface (318) wh.ch connect, to .^S^^g^S™ ^ the MCU 

the memory module. _ , . . . . 

The MCU attaches to the memory bus (MACD), ch *- ( ■ . . , rec eived by the MCU on the 
which provides the main communication path of the Memory ^^and follows the outgo- 
interconnect system. The MCU receives variable- SL^tomtwoM ^ f cIocU cycles . The 
length access requests at it memory bus mterface from a ■M£»J££ ^^""^jfied a t "WTT time as 
bus interface unit on the MACD bus and makes the 50 ^Z^S^SStoS^i of the' two cycles, 
proper series of accesses to memory through its storage he -gJ^^^SL? „der 20 data bits, and in the 
bus interface. After completion of the accesses to mem. *eMCU 8»° low order 12 data bils and 
ory. the MCU returns the proper reply on the MACD J«*g££^ e ^ the spare bit 

MACD bus is used for ell communication be- 55 JE^gSMM^Kfi 
.ween the MCU and other components. The access ^fc'^S J?or to doing the write so as to 

corresponding to these access types are handled on the been »i J read data 

memory bus: 
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node address and data is returned to the ECC unit. The array se- 
memory board type or ID quencer (515) controls ECC unit for error checking, 
MCU type or revision level while the data sequencer (515) is taking data back on the 
size of the attached memory array D-bus, and storing it in the reply data FIFO. The array 
refresh speed required 3 sequencer then signals the memory bus interface and 
memory access time in clock cycles receives a signal back from the memory bus interface 
connect to primary or backup bus. (Primary and called REPLY READY, 
backup bus operation is described in the above- Now the storage bus interface portion can start pro- 
referenced Budde et al application Ser. No. cessing another request if one's been queued up while 
296,025.) The MCU picks up the bus number from 10 the memory bus interface starts processing a reply when 
the specified memory bus during I NIT time. it gets a grant 
The MCU employs an error code in logic block (512) For a write request, data gets stored in the request 
that detects all single- and double-bit errors and cor- data FIFO (508), and the MACD bus is monitored by 
rects all single-bit errors. The MCU computes check the bus monitor (502) and the pipeline monitor (504). 
bits over the 32-bit data field, and also over the 20-bit 15 The opcode and address are stored in the message FIFO 
address field. Thus, the check bits represent not only a (509). When the presequencer (515) decodes the write 
unique data combination, but the unique data combina- request, it will start the array sequencer. The array 
tion at a unique address. Therefore, when reading the sequencer will first do a read, which obtains data from 
array, the MCU can detect an address line failure be- the memory array to check to see if the ECC indicates 
cause it will check ECC over the address that was sent 20 that this is the correct address. It then inserts the new 
to the array and the data that was received back. If the data from the data request FIFO (508) wherever it's 
data came from the wrong location, the check bits will supposed to go in the array. Then data is written out on 
indicate ah error in the failed address line. Therefore, the SLAD bus to the array with the ECC generated, 
reads are totally protected. Writes are also protected 

since they are always done as read-modify- writes to the 25 MESSAGES 
memory array. This means that a fully protected read What follows is a description of the messages that can 
precedes every write. After the complete memory ad- be sent over the memory bos between the bus interface 
dress is strobed into the RAMs, they become "locked" ' unit and MCU. The messages are separated into mes- 
on that address. Then the read is checked. If the read sage types: requests, replies, and bus blurbs, 
was error free, the address is known to be good, and the 30 dm 
write pulse is issued. If the read shows any error, the Request Messages 
operation is aborted before the write pulse is issued. Memory Read Request 

The MCU operates as follows. Each request on the Encoding: 0000LLLL HADR, LADR 
MACD bus will push the pipeline monitor (504) up one. Control Designation: 000,001 
When the pipeline monitor gets up to three deep or 35 This request is issued by a bus interface unit (BIU) to 
there are no more requests in the grant queue, the MCU an MCU in response to its attached processor's memo- 
knows that no bus interface unit can make a request, ry-read request. The receiving MCU satisfies the re- 
leaving the bus available for the MCU to make replies, quest by performing memory accesses and returning the 

The request comes in and goes first to the justify PLA ' following message: 

(JST PLA). The JST PLA asserts a decode signal to the 40 Read Reply 

message and reply FIFO (509) if the address falls within Memory RMW Read Request 

the module's address range, and then outputs a normal- Encoding: 0001LLLL HADR, LADR 

ized upper address. The JST PLA subtracts the starting Control Designation: 000, 001 

memory address from the requested address, normaliz- This request is issued by a bus interface unit to an 

ing it to zero, such that the memory-array address on 45 MCU in response to its attached processor's memory 

the storage array will always start at zero. The rest of RMW read request or as a result of a previously-queued 

the address goes into the message FIFO (509). memory RMW-read request propagating to the top of 

From the message FIFO the request is passed via a the bus interface unit's RMW queue. The receiving 

request latch to a presequencer in the sequencer logic MCU satisfies the request by performing memory ac- 

(515). The presequencer decodes the request and deter- 50 cesses and returning one of the following messagess: 

mines what kind of request it is and whether it's to be a Memory RMW Read Locked Reply 

register access, a diagnostic function, or memory ac- Memory RMW Read Reply 

cess. Memory Write Request 

If it is a diagnostic function, control is passed to the Encoding: 0010LLLL HADR, LADR, D0[, .... 

LCOM command sequencer in the sequencer logic 55 Dn] 

(515). If it is a memory access, control is passed to the Control Designation: 001, [. . . , 001,) 000, 001 

array sequencer in the sequencer logic (515). If it is a This request is issued by a bus interface unit to an 

diagnostic function that needs a memory access, the MCU in response to its attached processor's memory- 

LCOM will then pass control to the array sequencer write request The receiving MCU satisfies the request 

and the array sequencer will carry out the memory 60 by performing memory accesses and returning the fol- 

access. The array sequencer sequences the storage array lowing message: 

controls and controls data flow in and out of the SLAD write Acknowledge 

Bus Interface and ECC unit (518) to the storage array. Memory RMW Write Request 

A data sequencer in the sequencer logic (515) receives Encoding: 0011LLLL HADR. LADR, D0[ 

control from the array sequencer, and it controls data 65 Dn) 

from the ECC unit back to the data reply FIFO (510). Control Designation: 001, [...,, 001.) 000, 001 

For example, consider how this logic handles a read ■ This request is issued by a bus interface unit to an 

request. First, an address is placed on the SLAD bus MCU in response to its attached processor's memory 
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RMW-write request The receiving MCU satisfies the The response of the bus interface unit is to allow the 

request by performing memory accesses and returning processor to continue. 

the following message: Memory RMW Read Acknowledge 

Memory RMW Write Acknowledge Encoding: DO, Idle 

Interconnect Register Read Request 5 Control Designation: 010, 101 

Encoding: 0100000L HADR, LADR or 

Control Designation: 000, 001 Encoding: DO. Dl[, . . . , Dn] 

This request is issued by a bus interface unit (whose Control Designation: [011, . . . , 011.J 010, 101 

bus number matches the specified bus number in the This reply is the normal reply issued in response to 

address to an MCU or a bus interface unit whose rood- 10 the following request: 

ule number matches the specified module number) in MCU to bus interface unit— Memory RMW Read 
response to its attached processor's interconnect regis- Request 

ter-read request The receiving MCU/bus interface unit The response of th bus interface unit is to allow the 
satisfies the request by performing the access and re- processor to continue, 
turning the following message: 15 Memory RMW Write Acknowledge 

Read Reply Encoding: 11111111 11111110, Idle 

Interconnect Register Write Request Control Designation: 110, 111 

Encoding: OllOOOOL HADR, LADR, DO This reply is the normal reply issued by an MCU to a 

Control Designation: 001, 000, 001 bus interface unit in response to a memory RMW-wnte 

This request is issued by a bus interface unit in re- 20 request. The response of the bus interface unit is to 
sponse to its attached processor's interconnect register- allow the processor to continue. Other bus interface 
write request (that is, the bus interface unit whose bus units pop their RMW queues since this indicates the 
number matches the specified bus number in the address normal completion of an RMW access, 
to an MCU or a bus interface unit whose module num- Memory RMW Read Locked Reply 
ber matches the specified module number). The receiv- 25 Encoding: 11111111 11111100, Idle 
ing MCU/bus interface unit satisfies the request by Control Designation: 110, 111 
performing the access and returning the following mes- This reply is issued by an MCU to a bus interface unit 
J™. in response to a memory RMW-read request if the 

Write Acknowledge MCU is already locked by another processor's RMW 

Force Bad ECC (FBE) Request 30 request The bus interface unit queues the request on its 

Encoding- 1010LLLL HADR, LADR RMW queue and issues an RMW enqueue message so 

Control Designation: 000, 001 that other bus interface units will queue this request on 

This request is issued by a bus interface unit to an their RMW queues. 
MCU in response to a permanent module error if the Bus B\urbs 

request was a multimodule write and the error-log mod- 35 
ule ID matches but the error-log bus ID does not IPC 

match. {Error logs are described in the abovereferenced Encoding: 00000000 PPPPPPPP, Idle 
Budde, et al U.S. Pat. No. 4,438,494 ). The receiving Control Designation: 100, 111 
MCU satisfies the request by inverting the ECC bits in RMW Enqueue 
any 32-bit word which contains one or more bytes of 40 Encoding: 11111111 11111111, Idle 
data indicated in the LLLL field of the request It then Control Designation: 100, 111 
returns the following message: MACD BUS ARBITRATION MECHANISMS 

Write Acknowledge 

A large number of processors may be attached to a 
Reply Messages 45 B fajg c MACD bus by means of-bus interface units. Arbi- 

Read Reply tration musi be provided between contending requests 

Encoding- DO Idle for access to the bus. The arbitration mechanism is more 

Control Desimanon: 010, 111 fully described io the above-referenced copending ap- 

^ uci 6 plication Ser. No. 342,837 of David Budde et al. This 

i • „i :- %.M«n*t /<«/<nK«H tw>1rt\u 



or 



Encoding- DO Dlf, . . . , Dn] 50 arbitration mechanism is briefly described below. 

Control Designation: [011, . . . , 011,] 010, 011 The arbitration logic of FIG. 2, that coordinates the 

This reply is the normal reply issued in response to traffic of requests from these processors and the return 

one of the following requests: replies meets the following cntera: 

MCU to bus interface unit— Memory Read Request No processor request can be starved (i.e., never 

MCU or bus interface unit to bus interface unit— In- 55 serviced). 

terconnect Register Read Request A minimum number of input/output pins is required. 
The response of the bus interface unit is to take in the Bus pipeline and request-reply pamng is used to ad- 
data and return it to its attached processor. vantage. 

Write Acknowledge The number of "dead cycles" on the MACD bus 

Encoding: 11111111 11111111, Idle 60 caused by waiting for arbitration is minimized. 

Control Designation: 110, 111 The arbitration mechanism is fault tolerant. 

This reply is Ihe normal reply issued in response to Referring to FIG. 2, requests received from proces- 

™- „f *tJ i iwiwina r*miMt«- sors go through two levels of arbitration before going 

^^ZT^^M^ Write Request. onto'he Ma!d bus. T^e first ordering of requests is 

MCU or bus interface unit to bus interface unit-In- 65 done via a first^me, first-served a gonthm within the 

ter^nnect Register Write Request - time-ordered queue (408). The time-ordered queue 

MCU to bus interface unit-^Force Bad ECC Re- . keeps requests separated into time-order slots and as- 

quest sures that all requests in a particular time^order slot arc 
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serviced before any requests in the nexr time-order slot. additional time penalty paid in those cases for doing 

This guarantees that starvation will never occur. binary arbitration. 

It is possible, however, that several processor re- When contention between several requests does arise, 
quests may be made at the exact same time and be as- binary arbitration .starts narrowing down contenders 
signed to the same time-order slot in the time-ordered 5 until a winner is found. To accomplish this, each bus 
queue. For this reason a second level of arbitration interface unit utilizes its unique 6-bit logical module 
orders multiple requests within a single time-order slot. number. The binary arbitration slate machine (410) 
The algorithm used is found within the arbiter machine asserts RQOUT when it reaches ARBREQ from the 
(410) and is called binary arbitration. Based on the logi* time-order queue (408) and examines RQ# (Request), 
cal module numbers of the bus interface units making »0 an input coming from an external or-tied line driven by 
the requests, the arbiter machine (410) separates and the RQOUTs along the bus, (exactly like NREQ# and 
orders requests from the time-ordered queue and places NREQOUT). RQOUT (request out), is an output as- 
them in a three-deep grant queue (412). At this point serted by the bus interface unit binary arbitration ma- 
requests are fully ordered and only one request is at the chine (410) to indicate to other bus interface units that it 
head of the grant queue at any one time. 15 »* contending for access to the bus. The binary arbitra- 
Since requests and replies always come in pairs, the "on state machine (410) examines CONT0 (contention, 
ordering of replies is avoided by stipulating that replies an input which is pulled down upon the condition that 
must return in the same order as their corresponding two or more RQOUT lines are asserted, signifying that 
requests were made. A request FIFO (508) in the MCU two or more bus interface units are contending for ac- 
stores requests in the order that they are made. In the 20 cess to the bus. Together RQ# and CONT# having the 
bus interface unit, a pipeline-monitor machine (414) following meaning: 

makes sure that replies from the MCU are returned to 

the processor in the correct order. RQ# CO nt# M EANING 

Both requests and replies are fully ordered, such that „ ; No one is making a request '. 

at any one time there is only one request and one reply 0 , bus j^ffe^ ^ a m *kins t request aod it 

that can go onto the MACD bus. The MACD bus pro- ge i$ & grant 

tOCOl is designed tO Work in conjunction With the arbi- 0 0 Two or more but bterftce uniu are contcoding. 

tration mechanism so that there is never contention 

between the request at the top of the grant queue (412) 3() -r^e fourth combination can never validly occur, 

and the reply at the top of the pipeline (414). This proto- when two or more bus interface units contend, binary 

col only allows requests at certain times and replies at arbitration may take a number of cycles to determine 

other times. At no time does the protocol allow both a which bus interface unit will be granted access to the 

request and reply onto the bus at the same time. bus. At each successive cycle, each bus interface unit's 

Time ordering is the first stage of arbitration for re- 33 binary arbiter machine scans the next bit in its respec- 

quests coming from processors. Assuming that there are t j vc logical module number to make a decision as to 

31 processors that can make requests, each bus interface whether it should continue to arbitrate or concede to 

unit requires two pins and a 1 -bit by 3 1 -deep time-order- another bus interface unit with a higher priority logical 

ing queue (408) to implement time ordering. MCUs do module number. Logical module numbers are scanned 

not have time ordering since they never make requests, 40 f r0 m most significant bit (MSB) to least significant bit 

they only make replies. (LSB) and at each bit position, bus interface units with 

When a bus interface unit receives a request from its zeros \ n their numbers win out over those with ones, 

processor, the bus interface unit asserts the pin NREQ- modules (both bus interface units and MCUs) 

OUT (New Request Out). External to the bus interface maintain the grant queue in order to facilitate MACD 

unit, NREQOUT drives the input of an open collector 45 bus protocol which will be described subsequently, 

inverter (411) whose output is or-tied to the NREQ# when all of the requests at the top of the respective 

line. time-ordered queues in the bus interface units have been 

NREQ# goes low (becomes asserted) any time a new gj VC n grants by the arbiter, each bus interface unit's 
request is received by any bus interface unit along the arbiter generates an internal signal called BACOM (Bi- 
MACD bus. With respect to the time-ordered queue 50 nary Arbitration COMplete). BACOM causes the time- 
(408), every cycle that NREQ0 is low defines a new ordered queue to pop and introduces the next time- 
time-order slot. When a bus interface unit sees that order slot to begin binary arbitration. 
NREQrV is asserted, it pushes a one onto the time- The grant queue (412) is a three-deep FIFO which is 
ordered queue if it just asserted NREQOUT; otherwise, used to buffer granted requests waiting for bus time, 
it pushes a zero onto the time-ordered queue. All bus 55 Unlike the time-ordered queue, the grant queue has 
interface units in a row on the MACD bus push their only one request at the head of the queue in one of the 
respective time-ordered queue at the same time, pushing bus interface units at any one time. The request at the 
a one if they have a request in that particular time-order head of the grant queue is guaranteed to be the next 
slot, and a zero if they do not. The queues thusly oper- request to go out onto the MACD bus. After the request 
ate in parallel. 60 has been made, then it is popped off of the queue. The 

The bit at the head of each of the time-ordered queues purpose of the grant queue is to allow the arbiter to 

is called ARBREQ (Binary Arbitration Request) and it proceed independently of and in parallel with the bus 

indicates to the binary arbiter (410) that the bus inter- protocol and to get as much as three grants ahead of the 

face unit wants to participate in binary arbitration. The bus. As long as the arbiter can generate grants as fast or 
majority of the time there will only be a single request 65 faster than the bus can use them, all of the internal de- 

per time-order slot and in those noncontended cases, lays due to both time ordering and binary arbitration 

binary arbitration is extremely simple, a grant is issued will not introduce delays or dead cycles onto the 

to that request immediately. There is absolutely no MACD bus. While, the bus is busy with a particular 



request, the arbiter is able to work in parallel on gener- deep grant queue has been met. The MACD bus moni- 
ating a grant for a request to.be put on the bus at a future tor signals the message controller via the BUSY line if 
time. The bus rarely needs to wait for a grant. the MACD bus is presently occupied with a request or 

reply message. Even though the MCU does not send 
MACD Bus Protocol 5 re q Uest messages, it maintains a grant queue in order to 

The MACD bus is used for both request and reply implement the above-described protocol, 
messages. While an MCTTdoes ; „ot send request mes- MEMORY ACCESS PROCEDURE 

sages, it must contend for MACD bus access along with 

the bus interface units in order to place a reply message Memory accesses are received from.a processor by all 
on the bus. Therefore an arbitration mechanism without 10 bus interface unit's in a column over the ACD bus, 
a time-ordered queue is implemented in the MCU. The strobed by ISA, in accordance with the ACD bus pro- 
grant decoder and grant queue (506) guarantees that tocol described in the above-referenced Jackson apph- 
there will only be one bus access request with a grant at cation. When an access is detected, a bus interface unit 
the top of the grant queue at any one time. Likewise, the will perform the following operations: 
pipeline monitor guarantees that there is only one reply 15 I. Access Recognition: determine participation, refer- 
that can be the next reply on the bus. In order to pro- ring to memory starting and ending addresses and 
vide that the request at the top of the grant queue and to the interleave type and value: 
the reply at the top of the pipeline do not have to arbi- 2. Arbitration: arbitrate for access to the memory bus; 
trate with each other for use of the MACD bus (wasting 3. Grant: issue a memory bus access, with appropriate 
precious bus cycles) the following protocol causes the 20 address and length, as required to perform its part 
bus to be shared between requests and replies and maxi- of the processor bus request; 
mixes the bandwidth of the bus. This protocol is imple- 4. Release: release memory bus to further requesters; 
men ted in every bus interface unit in the messaae con- 5. Wait for response; 
troller machine (416) and every MCU in the message 6. Accept reply, as required. 

controller machine (500). The protocol guarantees that 25 Participation in an access is recognized in the bus 
any time at most one bus interface unit/MCU will drive interface logic (402) by checking portions of the re- 
t h e 1,^. ceived ACD address against the following information: 



Condition* 



BUSCNT MYTURN GNTQMT PIPEFUL BUSY Action 



_ i J — o Put reply on the bus. 

_ i o l 0 Put reply oa the bus. 

\ — — 0 0 Put request or blurb oa 

the bua. 

BU5GNT: I Ancrtcd by pant queue (412) If this bus Interface unit haa ■ request ml the top of the grant 

WYTTURN: 1 Aiaerted by the pipeline monitor U" IhU bm biiaUct unit Of Orfl MCU b the next in turn, u 
fodicitrd by the pipeline, to nuke a reply. 

GNTQMT: 1 Averted by the grant qoeue if «oy reqoeawhave received grant* and ere walling for the bin. 
PIPEFVt: 1 Averted by pipeline monitor if there ire three reqaeate which have been made that are waiting 

BUSY?^Afaertod by MACD bra b* the MACD bui n presently occupied with a request or reply menage. 

The latency between a request and its reply during 

heavy bus usage is the length of approximately four memory cluster start, memory cluster end, interleave 

messages. The time can be used by the MCU to access type and value. Computations using starting and ending 

the memory storage array and prepare a reply. 45 cluster values allow a bus interface unit to Find ir a 

request should be passed to the memory bus. Detcrmin- 
Request FIFO, Pipe Monitor, and Reply Monitor ing part i c i pa tion causes indication of a new time-order- 
The request FIFO (508) in the MCU, and a request ing cycle to other bus interface unit components on the 
FIFO within the message controller (416) in the bus memory bus, using NREQOUT. 
interface unit, is where request messages are queued by $0 Sharing of the memory bus among the bus interface 
the replier (the MCU or another bus interface unit, for units attached to that memory bus is determined by a 
those cases where the request is directed to a bus inter- two-stage arbitration scheme which guarantees cyclical 
face unit) while a reply message is being prepared. The sharing of the bus with no possibility for lockout or a 
message-controller machine in the MCU or bus inter- requester. The first stage is m""^*-^ 
face unit uses the request message in formulating the 55 first served — and is implemented as a FIFO queue (408). 
reply A request message is popped off the request Since any number of the processors on a memory bus 
FIFO when the matching reply has been put out onto may request access during any one time-ordering cycle, 
the bus. The reply monitors keep track of the depth of a second level of arbitration is provided. This arbitra- 
ge pipe as well as the ordering of replies. The reply tion (410) is a serial priority scheme. From a global 
monitor signals via the MYTURN line when it is the 60 standpoint, the accesses are time ordered. Priority 
replying bus interface unit/MCUs turn to use the bus under the time-ordered arbitration is determined by a 
for the purpose of making a reply. The reply monitor of binary sort in the 6-bit node address of the bus user The 
the oririnal requester signals via the MYRPY line when sort is accomplished by all requesters pulling the bus 
the nexTreply on the bus is the one in response to the request line (RQOUT). It it is determined that there « 
request of that original requester. The pipeline monitors 65 no conflict (only one requester) then bus access s 
signal the message controllers via the PIPEFUL line if granted. If there is a request conflict (multiple request- 
there are three requests which have been made that are ers) the sort begins. This is a sequential process taking 
waiting for replies, thus signifying that the limit of the 3 up to six cycles. In the first cycle, all requesters whose 
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high order node address bit is zero, remove their re- 
quest from the bus request line. If the conflict is then 
resolved, the arbitration is complete; if not the arbitra- 
tion proceeds to the next bit in the node address, and so 
on until there is no conflict The winner of the arbitra- 5 
tion then proceeds, and the other requesters compete 
for bus access in the next arbitration cycle. 

Contention is determined by an or- tied (CONT#) 
line which is pulled high by a constant current source. 
When a bus user pulls the bus request line (RQOUT), it 10 
also pulls this contention line through its own constant 
current source. The current sources are adjusted so that 
two or more will deplete the current source pulling the 
CONT# line high, thus the voltage level will fall. 

When a bus interface unit completes recognition of an 15 
access, a new time ordering cycle is signaled to other 
bus interface units attached to the same memory bus by 
asserting NREQOUT. (Other bus interface units on that 
memory bus may randomly recognize accesses from 
their own attached processor at the same time. These 20 
requests all belong to the same time ordering-cycle, 
signaled simultaneously on NREQ# by all requesting 
bus interface units using NREQOUT.) 

Indivisible read-then-write access pairs to a data item 
are supported by the time-ordered arbitration mecha- 25 
nisnv A bus interface unit accommodates this access by 
participating in a read-modify-write (RMW) locking 
protocol with other bus interface units and the memory 
unit on the memory bus. 

There are two phases of RMW activity. When a 30 
processor makes an RMW request, it first traverses the 
normal time-ordering queue as described above. When 
the time cycle arrives for service of the request, there 
are two possibilities: (1) the memory bus is locked to 
RMWs, and (2) it is OK to attempt an RMW-read re- 35 
quest. 

If the memory bus is not locked, a bus interface unit 
with an RMW-read will simply go ahead with its re- 
quest. If an RMW operation is underway, from another 
bus interface units RMW-read, and there has already 40 
been an intervening attempt to do another RMW read, . 
which failed, the RMW which is locked out is queued in 
a secondary time-ordering queue. Time-ordered service 
of the RMWs is accomplished by enqueuing the locked 
RMWs in the order they reacrrthe-top of the normal 45 
time-ordering FIFO. When an RMW-write completes 
the pending RMW, those bus interface units with en- 
queued RMW reads will take their turn requesting 
RMW-reads. After an RMW read is successfully ser- 
viced for a bus interface unit, it may then continue 50 
accepting norma) reads and writes from its attached 
processor. 
The protocol outlined above assures: 
RMW service in time order with other RMWs, 
RMW service as soon as possible, 55 
Continuation of normal requests* throughput 
While the invention has been particularly shown and 
described with reference to preferred embodiments . 
thereof, it will be understood by those skilled in the art 
that the foregoing and other changes in form and detail 60 
may be made therein without departing from the spirit 
and scope of the invention. 
What is claimed is: 

1. For use with a memory bus including a data por- 
tion, said bus providing the means of communication 65 
between a memory control unit connectable to a mem- 
ory module capable of being accessed by said memory 
control unii, and at least one bus interface unit connect- 
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able to a data processor, said data processor being capa- 
ble of issuing memory requests including address infor- 
mation for a data transfer operation requested of said 
memory and capable of receiving memory replies in- 
cluding data requested from said memory, each of said 
memory replies being responsive to a particular mem- 
ory request, wherein accesses to said memory are han- 
dled by means of a series of messages transmitted to said 
memory control unit in accordance with a specific con- 
trol protocol, said data processor being further capable 
of issuing control information for bus transactions, the 
combination comprising: 
message generator means for generating messages in 
the form of packets of information for transmission 
on said memory bus, said messages being divided . 
into message types including control message types 
of packets, request message types of packets corre- 
sponding to said memory requests, and reply mes- 
sage types of packets corresponding to said mem- 
ory replies, each packet comprising one or more 
bus transmission slots issued by said message gener- 
ator means sequentially and contiguously, each bus 
transmission slot in a packet being capable of in- 
cluding an opcode, address, data, control, and pari- 
ty-check bits; 
a pipeline queue; 

message controller means connected to said message 
generator means and to said pipeline queue for 
controlling said request message packets and said 
reply message packets such that a predetermined 
number of said request message packets may be 
entered into said pipeline queue at any one time; 

monitor means connected to said message controller 
means and to said memory bus for monitoring said 
request message packets and said reply message 
packets generated on said bus by said message gen- 
erator, such that request message packets in excess 
of said predetermined number are prevented from 
being generated until a reply message packet is 
received to thereby free-up a slot in the pipeline; 

control signal lines operative in parallel with the data 
portion of said bus for providing a coded signal 
representing a particular message type generated 
on said bus by said message generator; and, 

interface logic means connected to said message gen- 
erator means and to said control signal lines, re- 
sponsive to said message generator, for driving said 
control lines to indicate the message type; 

said message generator means including means for 
inserting a particular reply message packet corre- 
sponding to a particular request message packet in 
said pipeline queue at a position in said pipeline 
queue corresponding to the request message packet 
that is associated with said particular reply message 
packet. 

2. In a data processing system in which a switching 
matrix provides electrical interconnections between 
horizontal MACD buses and vertical ACD buses con- 
nected in said matrix by means of nodes including bus 
interface unit nodes and memory-control unit (MCU) 
nodes, communication means in one of said nodes com- 
prising: 

recognizing means connected to said ACD bus for 
recognizing signals on said ACD bus representa- 
tive of a request to said one node; 

means connected to said MACD bus, responsive to 
said recognizing means, for generating on said 
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MACD bus, signals on said MACD bus representa- 
tive of a write-request packet or a read-request 
packet, said packets comprising a number of bus 
transmission slots, issued sequentially and contigu- 
ously, each slot in a packet including opcode/ad- 
dress/data bits and control bits, 

said write-request packet comprising an opcode spec- 
ifying a function desired, an address specifying a 
physical memory location, and write data, 

said read-request packet including an opcode specify- 
ing a function desired, and an address specifying a 
physical memory location; 

receiving means connected to said MACD bus for 
receiving a reply message packet on said MACD 
bus, said reply message packet comprising a num- 
ber of bus transmission slots, received sequentially 
and contiguously, each slot in said reply packet 
including opcode/data bits and control bits, 

said reply message packet containing a header op- 
code and/or data; 

message control means connected to said generating 
means and said receiving means, said message con- 
trol means including: 

means for queuing data in said write request packet, 
means responsive to said control bits in said read M 
and write packets for detecting the start and end of 
a packet, means for decoding said opcode to per- 
form a function desired, 

means for establishing a pipeline for said request mes- 
sage packets and said reply message packets gener 
ated on said bus by said message generator, and, 

monitor means connected to said message control 
means and to said MACD bus, for monitoring sig- 
nals on said MACD bus representative of said re- 
quest and reply packets on said bus, such thai a 
request packet is prevented from entering said 
pipeline until a reply packet is received to free-up a 
slot in said pipeline, whereby request packets in- 
crease the length of said pipeline up to a predeter- 
mined number of request 

packets in said pipeline, and reply packets decrease 40 
the length of said pipeline; and, 

means connected to said recognizing means, said 
generating means, said message control means, and 
said receiving means, for arbitrating said signals on 
said ACD bus representative of requests from said 45 
ACD bus going onto said MACD bus and said 
signals on said MACD bus representative of replies 
received from said MACD bus, to thereby enable 
said message control means to insert a reply packet 
associated with a particular request packet in the 30 
pipeline position of said recognized request packet 

3. The combination in accordance witn claims 1 or 2, 
wherein said monitor means includes: 

reply monitor means including means for maintaining 
a queue of reply packets in the same order as the 55 
order of associated request packets, means for in- 
forming a replying node when it is time to reply 
and means for informing a requesting node when 
an associated reply packet is to arrive. 

4. The combination in accordance with claim 1 fur- 60 
ther comprising: 

means connected to said monitor means and said 
message controller means, for arbitrating said sig- 
nals on said ACD bus representative of requests 
from said ACD bus and said signals on said MACD 65 
bus representative of replies received from said 
MACD bus, to thereby enable said message con- 
troller means to insert a reply packet associated 
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with a particular request packet in the pipeline 
position of said request packet 
.5. The combination in accordance with claim 1 
wherein said means for generating packets includes: 
means for generating on said bus a write-request 

packet and a read-request packet, 
said write-request packet comprising an opcode spec- 
ifying a function desired, an address specifying a 
physical memory location, and write data, 
said read -request packet including an opcode specify- 
ing a function desired, and an address specifying a 
physical memory location; and, 
wherein said message controller means further com- 
prises: 

means for queuing data in said write request packet; 
means responsive to said control bits in said read and 
write packets for detecting the start and end of a 
packet; and, 

means for decoding said opcode to provide an indica- 
tion of a function. 
6. The combination in accordance with claim 3 fur- 
ther comprising: 

means connected to said monitor means and said 
message controller means, for arbitrating said re- 
quests onto said bus and said replies received from 
said bus, to thereby enable said message controller 
means to insert a reply to a particular request in the 
pipeline queue at a position in said pipeline queue 
corresponding to the request message packet that is 
associated with said particular reply message 
packet. 

The combination in accordance with claim 3 



7. 

wherein said means for generating packets includes: 

means for generating on said bus a write-request 
packet and a read-request packet, 

said write-request packet comprising an opcode spec- 
ifying a function desired, an address specifying a 
physical memory location, and write data, 

said read-request packet including an opcode specify- 
ing a function desired, and an address specifying a 
physical memory location; and, 

wherein said message controller means further com- 
prises: 

means for queuing data in said write request packet; 
means responsive to said control bits in said read and 

write packets for detecting the start and end of a 

packet; and, 

. means for decoding said opcode to provide an indica- 
tion of a function. 
8. The combination in accordance with claim 4 
wherein said means for generating packets includes: 
means for generating on said bus a write-request 

packet and a read-request packet, 
said write-request packet comprising an opcode spec- 
ifying a function desired, an address specifying a 
physical memory location, and write data, 
said read-request packet including an opcode specify- 
ing a function desired, and an address specifying a 
physical memory location; and, 
wherein said message controller means further com- 
prises: 

means for queuing data in said write request packet; 
means responsive to said control bits in said read and 
write packets for detecting the start and end of a 
packet; and, 

means for decoding said opcode to provide an indica- 
tion of a function. 



